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Abstract. Generating trees are a useful technique in the enumeration of various combinato- 
rial objects, particularly restricted permutations. Quite often the generating tree for the set of 
permutations avoiding a set of patterns requires infinitely many labels. Sometimes, however, 
this generating tree needs only finitely many labels. We characterize the finite sets of patterns 
for which this phenomenon occurs. We also present an algorithm — in fact, a special case of 
an algorithm of Zeilberger — that is guaranteed to find such a generating tree if it exists. 



1. Introduction 

The standardization of a word w consisting of k distinct integers is the permutation st{w) 
of length k obtained by replacing the smallest entry of w; by 1, the second smallest entry by 2, 
and so on. If /3 G S^, we say that the length n word of distinct integers w contains a (3-pattern 
if and only if it contains a (not necessarily contiguous) subword whose standardization is (5. 
Otherwise we say that w is /3-avoiding. For example, the permutation 239145678 contains a 
312-pattern (for example, the subword 916 standardizes to 312) but avoids 321. If i? is a set 
of permutations, we say that the word w avoids B if it avoids every member of B. Such a set 
B is often referred to as a set of forbidden patterns. Although we have allowed w to be a word 
in our definition, we are most interested in the case where w is a, permutation. Let Av„(i?) 
denote the set of B-avoiding permutations of length n and let Av(i?) denote the set of all finite 
i?-avoiding permutations. We refer to | Av„(i?)|a;" as the generating function for Av(S). 

The problem of enumerating restricted permutations has received considerable attention 
over the last two decades. Bona's book [8] provides an overview of these efforts. A common 
technique is that of generating trees, introduced by Chung, Graham, Hoggatt, and Kleiman [12] 
and used extensively by many others^ since. The recently introduced ECO (enumerating 
combinatorial objects) method^ extends the notion of generating trees to other combinatorial 
contexts. There has also been some interest in the algebraic properties of generating trees and 
ECO systems^. 

Precisely, a generating tree is a rooted, labeled, and typically infinite tree such that the label 
of a node determines the labels of its children. Sometimes the labels of the tree are taken to be 
natural numbers, but this is not necessary, and the algorithm we will describe labels nodes by 
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Figure 1. The first four levels of the pattern-avoidance tree r(132,3241) 

permutations. Therefore we specify a generating tree by supplying the label of the root and a 
set of succession rules. For example, the complete binary tree may be given by 

Root: {SS) 

Rule: (SS) ^ {SS){SS). 

The connection to restricted permutations comes through pattern- avoidance trees. We say 
that the permutation a of length n is a child of vr € Sn-i if cr can be obtained by inserting n 
into IT. This defines a rooted tree T on the set of all permutations. For a set of patterns B, we 
define the pattern-avoidance tree T{B) to be the subtree of T whose nodes are the i?-avoiding 
permutations Av(B). For example, the first four levels of r(132,3241) are shown in Figure 1. 
The active sites (relative to of vr € Av„_i(i3) are the positions i for which inserting n right 
before the ith. entry of vr produces a S-avoiding permutation. By convention, n + 1 is an active 
site of vr if appending n to the end of vr produces a S-avoiding permutation. An inactive site 
is any site that is not active. For example, the active sites of 213 relative to B = {132,3241} 
are 1 and 4, whereas the inactive sites are 2 and 3. 

Given a pattern-avoidance tree T{B), we would like to find an isomorphic (in the category 
of rooted trees) generating tree. For example, take a permutation vr G Av„(132, 3241). There 
are at most three sites in which we may insert n + 1 to form a {132, 3241}-avoiding child: the 
beginning, the end, and the site directly to the right of n. Indeed, if we insert n + 1 to the 
left of n but not at the beginning we form a 132-pattern, while if we insert n + 1 further than 
one site to the right of n but not at the end we get a subsequence n, x, {n + l),y and either 
X < y, giving a 132-pattern, or x > y, giving a 3241-pattern. We can therefore constuct an 
isomorphic generating tree with just two labels, label (2) for the nodes where n is the last 
entry and label (3) for the nodes where n is not the last entry. The generating tree is then 

Root: (2) 

Rules: (2) (2) (3) 

(3) - (2)(3)(3). 

On the other hand, the pattern-avoidance tree T(123) requires infinitely many labels; by 
considering the lexicographically first ascent in a 123-avoiding permutation one can show that 
r(123) is isomorphic to the generating tree given by 

Root: (2) 

Rules: (j) ^ (2)(3) • • • (j + 1). 

Upon finding a generating tree isomorphic to T{B), one often wishes to get the gener- 
ating function for Av{B). In general, as witnessed by Bousquet-Melou [9] and Banderier, 
Bousquet-Melou, Denise, Flajolet, Gardy, and Gouyou-Beauchamps [3], this process can be 
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quite intricate. However, if the generating tree has only finitely many labels then the gener- 
ating function for Av(-B), which must be rational, can be computed using the transfer matrix 
method (see Stanley's text [32, Section 4.7] for details). 

Herein we characterize the finite sets B for which T{B) is isomorphic to a finitely labeled 
generating tree, answering a question raised earlier [34]. One requirement for T[B) to be 
isomorphic to a finitely labeled generating tree is that there must be a bound on the number 
of children a node of T{B) may have. For this to occur, B must contain both a child of an 
increasing permutation (such as 132, 4123, or 12345) and a child of a decreasing permutation 
(such as 231, 3241, or 54321), because otherwise either 12- --n or n---21 will have n + 1 
children for all n. In fact, Kremer and Shiu [22] showed that this is enough. We include a 
short proof below. First recall the following famous theorem of Erdos and Szekeres. 

Theorem 1 (Erdos and Szekeres [16]). Every permutation of length at least (A; — 1)(^ — 1) + 1 
contains either an increasing sequence of length k or a decreasing sequence of length i. 

Theorem 2 (Kremer and Shiu [22]). The pattern- avoidance tree T{B) has bounded degrees if 
and only if B contains both a child of an increasing permutation and a child of a decreasing 
permutation. 

Proof. We have already noted that the condition on B is necessary, so it suffices to show that it 
is sufficient. Assume not, so that although B contains a child of 12 • • • A; and a child of £ • • • 21, 
T{B) does not have bounded degrees. Set n = {k — — 1) + 1. 

We claim that there is a permutation tt G T{B) of length n with n + 1 children (or, equiva- 
lently, n+1 active sites). Since T{B) does not have bounded degrees, we can find a permutation 
in T[B) with at least n + 1 active sites. Suppose that ii < 12 < ■ ■ ■ < in+i are active sites of 
TT. Now form the word w = vr(ii)7r(i2) • • • vr(z„) and set a = si{w). For example, suppose that 
n = 3 and vr = 461523 with active sites {1, 2, 5, 7}. Then we get w = 7r(l)7r(2)7r(5) = 462 and 
a = st(462) = 231. 

By construction, o" is a permutation of length n with n + 1 children in T(B), proving the 
claim. However, by our choice of n, the Erdos-Szekeres theorem shows that a contains either 
an increasing subsequence of length at least k or a decreasing subsequence of length at least i. 
Thus we have reached a contradiction because our assumptions on B imply that at least one 
of the children of a most contain a pattern from B and thus cannot be a node of T{B). □ 

In general, trees with bounded degrees need not be isomorphic to finitely labeled generating 
trees. For example, consider the generating tree (pictured in Figure 2) given by 

Root: (1,1) 

Rules: -w (i, j - 1) if j > 2, 

(i,l) ^ (i + i,i + i)(o), 

where nodes labeled by (0) do not produce children. This tree is clearly not isomorphic to a 
finitely labeled generating tree since the distance between two nodes of degree 2 is unbounded, 
but each of its nodes has at most two children. Our main result. Theorem 7, says that if B is 
finite and T{B) has bounded degrees then T[B) is isomorphic to a finitely labeled generating 
tree, so this example shows that our proof will need to make use of the special properties of 
pattern-avoidance trees. 

In the next section we introduce lemmas and notation for pattern-avoidance trees. Section 3 
describes our labeling algorithm, while the proof that this algorithm works is contained in 
Section 4. 

2. Removable & GT-reducible entries 

In order to motivate our technique we begin by returning to the example of T(132,3241). 
Since our approach in the last section was rather ad hoc, we now attempt to analyze this tree 
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Figure 2. A tree with bounded degrees that is not isomorphic to a finitely 
labeled generating tree 

(or rather, two of its nodes) in a more systematic manner. Consider 213. In this permutation 
we can insert new maximal entries at the beginning or the end, but we will never be able to 
insert a new maximal entry between the 2 and the 3. Now let vr denote a descendant of 213. 
From our previous comments, 213 appears as a contiguous block in vr. Observe that for any 
possible 132 or 3142-pattern in vr involving the 1 there is another pattern which uses the 2 
instead. Therefore it does not matter whether or not the 1 is stuck between the 2 and the 3, 
and we can assign the same label to 213 as we assign to 12. In fact, a similar argument shows 
that 12 can be labeled by the same label as 1 receives. It is notions like these that we aim to 
formalize in this section. 

First, we say that an entry x in the permutation vr is removable (relative to a set of patterns 
i?) if it is adjacent to at most one active site. For example, every entry of 213 is removable 
when B = {132,3241}, although no entry of 21 is removable. When more detail is needed, we 
say that the entry 7r(z) is left-removable if i is an inactive site of vr and right-removable if i -|- 1 
is inactive, so every removable entry is either left-removable, right-removable, or both. 

If X is an entry of the word w and w contains distinct integers then we write w — x to denote 
the word formed from w by removing x. For example, 461523 — 2 = 46153. If X is a set of 
entries oi w, we similarly write w — X to denote the word formed by removing each entry in 



When 71" is a node of T{B) we let T{B; vr) denote the subtree consisting of vr and its de- 
scendants. In order to avoid the shifting of indices and values caused by standardization we 
will also make use of the similar but less natural tree W{B; n, u), which we define whenever u 
is a S-avoiding word containing distinct integers all at most n. The root of W{B;n,u) is u, 
and it contains all S-avoiding words w that can be formed by shuffling u with a permutation 
on [n + l,n + \w\ — \u\]. If v and w are nodes of W{B;n,u) and w has greatest entry m, 
then w is a child of f if w can be obtained by inserting m into v, that is, if v = w — m. An 
example is shown in Figure 3. For a word w G W{B; n, u) we define active sites, inactive sites, 
removability, left-removability, and right-removability as we did for the permutations of T{B). 

Once these definitions are unraveled, it is evident that W{B]n,u) = T[B;st[u)) for all 
allowed values of n, and if tt is a permutation then W{B] |7r|,7r) and T{B;tt) are not merely 
isomorphic, but they are the same tree. 

If « is a word of length at least two containing the entry x we define the map 



X. 



: W{B- n, u) W{B; n,u-x) 



FINITELY LABELED GENERATING TREES AND RESTRICTED PERMUTATIONS 



5 



13 




13567 71356 61357 67135 76135 51367 75136 56137 56713 75613 65137 67513 76513 



Figure 3. The first four ievels of T^({132, 3241}; 4, 13) 

by dx{w) = w — X. Our upcoming Proposition 3 shows that dx is one-to-one if x is a removable 
entry in u. There are several different cases in the definition of the inverse map. 
First suppose that x = u{i) for some i > 2. In this case we define the map 

: ^(0; n,u-x) ^ VF(0; n, u) 

by letting denote the word obtained from w by inserting x immediately to the right 

of the entry u{i — 1). If x is left-removable in u relative to B then it is easy to see that l~x 
maps words in dx{W{B;n,u)) to words in W{B;n,u). Furthermore, if w; S W{B;n,u — x) \ 
dx(W{B;n,u)) then i'^xi'^) 'wiH contain at least one pattern from B. 

There are three more cases for us to define this operation. If x = n(l) then we simply let 
iu^xi"^) — Similarly, if x is the last entry of u we let Lu,xi''^) ~ Otherwise we let 
''u,xi'^) denote the word formed from w by inserting x to the immediate left of u{i + 1). 

Proposition 3. Let u be a B-avoiding word of length at least two containing distinct integers 
all at most n and suppose that x (z u is removable. Then dx ■ W{B; n, u) W{B; n,u — x) is 
one-to-one. More specifically, if x is left-removable then 

i^u,x ° 9x ■■ W{B; n, u) W{B; n, u) 

is the identity, and if x is right-removable then 

it,x°dx-W{B-n,u)^W{B-n,u) 

is the identity. 

Proof. As the various cases are quite similar, let us assume that x = u{i) for some i > 2 and 
that X is left-removable. Take w G W{B\n,u). Since x is left-removable, w cannot contain an 
entry between u{i — 1) and x. Thus applying dx removes x, but then inserts x immediately 
to the right of u{i — 1), restoring w. □ 

One of the implications of Proposition 3 is that W{B;n,u) embeds into W {B]n,u — x) by 
the map dx whenever x is a removable entry of u. It sometimes happens that these two trees 
are isomorphic. If W{B;n,u) = W{B;n,u — x), and x is removable, then we say that x is 
generating-tree-reducible relative to B, or for short, GT-reducible. This is a strengthening of 
Zeilberger's definition of reversely deleteable entries from [37]. In this language, our observation 
at the beginning of the section was that the entry 1 in 213 is GT-reducible for B = {132, 3241}. 
If the permutation vr contains a GT-reducible entry, we also refer to vr as being GT-reducible. 

Before ending our discussion of l, let us note that in many cases these maps commute: 

Remark 4. Let u be a word containing distinct integers all at most n, let x and y be nonad- 
jacent entries of u, and let (5, e G {+, — }. Then 
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as maps from VF(0; n,u — x — y) to W{^; n, u). 

We would now like to show that it is possible to decide whether a removable entry is GT- 
reducible. For this we need two more definitions. Given a tree T, let (.i{T) denote the number 
of nodes of T of height i, so io(T) = 1 unless T is the empty tree, ii(T) is the number of 
children of the root node, and ^2{T) is the number of grandchildren of the root node. Also, if 
B \s a, finite set of patterns, let ||-B||(X) denote the length of the longest pattern in B. 

Proposition 5. Let u he a B-avoiding word of length at least two containing distinct integers 
all at most n. The removable entry x ^ u is GT-reducihle if and only if 

lr{W{B] n, u)) = lr{W{B- n,u- x)) 

for alll<r < \\B\\oo - 1. 

Proof. If X G n is GT-reducible then W{B] n, u) = W[B; n, u—x) by definition, so (.r{W{B] n, u)) = 
ir{W{B;n,u — x)) for ah r G N. Suppose now that ir(W{B;n,u)) = ir{W{B;n,u — x)) for 
all 1 < r < ||i?||oo — 1- Since x is removable. Proposition 3 shows that W{B;n,u) embeds into 
W{B; n,u — x) by the map dx- Thus we would like to show that this map is onto. 

Suppose not and choose w G W{B; n,u — x) that is not in the image of dx- Let e = — if x is 
left-removable in u. Otherwise x must be right-removable in u, and here we let e = +. Since 
w ^ dx{W{B; n, u)), tu^xi"^) contains a permutation from B. Choose a subword of tu,a;('"^) that 
standardizes to a member of B and label it v. Because w avoids B, x must be an entry of v. 

Now consider the subword w' of il^^{w) containing all entries that are either in u or in v. 
Because w' contains -y, it contains a pattern from B. However, it;' — x is a subword of so 
it avoids B. We would like to find a word in ^.(^^(i?; n, u — x)) with these properties. We 
do this by "partially standardizing" w': replace the smallest entry of w' that is not in u by 
n -|- 1, the next smallest entry of w' that is not in ti by n -|- 2, and so on. Label the resulting 
word w" . Notice that w" G 3,(VF(-B; n,u — x)), w" contains a pattern from B, and w" — x is 
S-avoiding. These observations and Proposition 3 show that 

^\w"\-\u\{W{B;n,u)) < n, u - x)). 

Furthermore, since both w" and u contain x, \w"\ < |nj+||-B||oo — 1, contradicing our hypotheses. 

□ 

3. The algorithm 

Our work in the previous section suggests the following approach for finding a generating 
tree isomorphic to T{B). If 1 ^ Ky{B) then our task is quite easily accomplished, so let us 
assume that 1 avoids B. We start with a root node (1), a set P = {1} of permutations that 
we have not checked for GT-reducible entries, and a set 7^ = of succession rules. Now we 
pick a permutation tt G P of minimum length and check it for GT-reducible entries (we make 
the convention that the permutation 1 never has a GT-reducible entry). 

First suppose that vr is not GT-reducible, and say that its i?-avoiding children are (Ti, o"2, . . . , cJf . 
In this case we remove vr from P, add its S-avoiding children to P, and add the succession 
rule 

(vr) (cri)(o-2) • • • (dt) 

to 7^. 

If instead vr has a GT-reducible entry x then we again remove vr from P, but now we search 
through our set of succession rules 7^ and replace each instance of (vr) by the label we have 
given to the node st(vr — x) (this label might not be (st(vr — x)) because st(vr — x) may also 
have a GT-reducible entry). In other words, whenever a node labeled by (vr) would have been 
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produced, now a node labeled by the same label as st(7r — x) will be produced. This does not 
change the isomorphism type of the tree because 

T{B; vr) = W{B; |7r|, vr) ^ W{B; |^|, ^ - x) ^ T{B; st(7r - x)). 

We repeat this process until P = %. If we ever reach this state then we know that the 
generating tree we have produced is isomorphic to T{B). We will prove shortly (Theorem 7) 
that we do reach this state when B contains both a child of an increasing permutation and a 
child of a decreasing permutation. 

Before that, let us illustrate the process with the tree r(132,3241). We start with P = {1} 
and 7^ = 0. Then we choose 1 from P and note that it does not have a GT-reducible entry 
(by our convention), so we remove 1 from P and add 12 and 21 to P, giving us P = {12, 21}. 
We also add the rule 

(1) (12)(21) 

to our set of rules TZ. 

Now choose 12 from P. First we check to see if 1 is a GT-reducible entry. Since ||-B||oo = 4, 
Proposition 5 shows that we only need to test whether £^(W^({132, 3241}; 2, 2)) and £r(r({132, 3241}; 12)) 
agree for r = 1, 2, 3. The number are shown in the following chart. 



r 


£^(VF({132,3241};2,2)) 


4(T({132,3241};12)) 


1 


2 


2 


2 


5 


5 


3 


13 


13 



From this chart we may conclude that 1 is a GT-reducible entry, so we remove 12 from P and 
we replace (12) by (1) in all of our rules. After this we have P = {21} and IZ contains the 
single rule 

(1) (1)(21). 

We must now choose 21 from P. However, 21 does not have any removable entries, so it is 
not GT-reducible. We therefore remove 21 from P, add its children (321, 231, and 213) to P, 
and add the rule 

(21) (321)(231)(213) 

to TZ. 

Using the same process as with 12 we can find that both 321 and 231 have GT-reducible 
entries (the entry 2 is GT-reducible for both permutations), so we replace (321) and (231) in 
all our rules by (21). 

At this point we have P = {213}. This permutation also has a GT-reducible entry (again, 2) 
so if we were to follow the pattern of the previous cases we would simply replace all instances 
of (213) in our set of rules by (12). However, since 12 is GT-reducible itself we instead replace 
instances of (213) by (1). 

We are now done because P = 0, and we have found the generating tree 

Root: (1) 

Rule: (1) (1)(21) 

(21) - (21)(21)(1). 

Up to relabeling, this is the same tree we found in Section 1. Via the transfer matrix method, 
one can use these rules to compute the generating function for Av(132, 3241): 

1 Av„(132, 3241) |x" - "^^"^ ~ 



^—^ ^ ' — 3x + 1 ' 

n>l 

and thus the sequence | Av„(132, 3241)| contains every second Fibonacci number. 
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4. Proof of the main result 

It remains to show that this procedure terminates. In fact, if the nodes of T{B) have 
arbitrarily large degrees, then we will never reach a state where P = because in that case 
either 12 - ■ ■ n or n - ■ - 21 will never have a removable entry, let alone a GT-reducible entry. 
Our central result. Theorem 7 below, shows that if B is finite and T{B) has bounded degrees 
then this procedure will terminate. 

We begin with a technical lemma that will be used to construct large sets of entries satisfying 
a condition stronger than removability. 

Lemma 6. Suppose that B is a finite set of patterns containing both a child of an increasing 
permutation and a child of a decreasing permutation and fix a positive integer r. Every suffi- 
ciently long B-avoiding permutation vr contains a set X = {xi, . . . , Xr} of r distinct pairwise 
nonadjacent entries so that each xj is removable in the word vr — {X \ {xj}). 

Proof. We prove the lemma by induction on r. The base case r = 1 is immediate because 
Theorem 2 implies that all sufficiently long permutations contain removable entries, and we 
may take xi to be any removable entry in vr. 

Let vr be a i?-avoiding permutation of length n. The r = 2 case provides a nice illustration 
of our argument, so we examine it before moving on to the general case. Let yi, . . . ,ys denote 
the removable entries in vr — xi that are not adjacent to xi in vr, and assume to the contrary 
that xi is not removable in vr — for any i € [s]. Because xi is removable in vr, at least one of 
the sites adjacent to xi in vr is inactive. Now form the i?-containing permutation a by inserting 
n + 1 into vr in an inactive site adjacent to xi. Because xi is not removable in vr — y^, o" — yi is 
S-avoiding. But this means that every copy of a pattern from B in a must contain the entry 
yi and the entry n + 1, so s < \\B\\oo — 1. On the other hand. Theorem 2 shows that we may 
take s to be as large as we like so long as n is sufficiently large, a contradiction. 

Now suppose that r is any integer at least 2, and that we have found a set of entries 
{xi, . . . ,Xr-i} satisfying the desired conditions. We wish to find an entry Xr, not adjacent to 
any of the entries xi, . . . , Xj—i, so that {xi, . . . , x^} satisfies the desired conditions. As in the 
r = 2 case before, we begin by letting yi, . . . , denote the removable entries of vr— xi — • • •— x^-i 
that are not adjacent (in vr) to any Xj. Assume to the contrary that none of the y^'s will function 
adequately ctS Xf , SO for each i G [s] there is at least one j G [r — 1] such that xj is not removable 
in vr — xi — • • • — xj-i — Xj+i — • • • — Xr-i — yi- Choose one of these values to be denoted j{i). 

When we were trying to find an entry to serve as X2 we built a single permutation a. This 
time we need to consider the r — 1 permutations ai, . . . , ctj—i where aj is formed by inserting 
n + 1 into vr in an inactive site adjacent to Xj and then removing xi, . . . , Xj_i, Xj+i, . . . , Xr-i- 
Each of these permtuations contains a pattern from B and by our assumptions, fTj(j) — yi avoids 
B for each z G [s]. As before, we now ask how many different values of i can share the same 
value j{i). The answer is the same: every copy of a pattern from B in (Tj(j) must contain both 
yi and n + 1, so at most |Ii?||oo — 1 values of i may share the same Therefore we need 

only have s > (r — l)(||i?|loo — 1), which we get if n is sufficiently large, to guarantee that at 
least one of the y^'s can serve as Xr, completing the proof of the claim. □ 

Theorem 7. Let B be a finite set of patterns. The pattern- avoidance tree T{B) is isomorphic 
to a finitely labeled generating tree if and only if B contains both a child of an increasing 
permutation and a child of a decreasing permutation. Furthermore, if T{B) satisfies these 
conditions then the algorithm presented in Section 3 will find a finitely labeled generating tree 
isomorphic to T[B). 

Proof. These conditions on B are necessary by Theorem 2. To prove the other direction, it 
suffices to show that every sufficiently long permutation is GT-reducible. Take vr to be a B- 
avoiding permutation of length n. We prove the theorem by showing that if r is sufficiently 
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large then at least one of the removable entries xi, . . . ,Xr guaranteed by Lemma 6 must be 
GT-reducible. 

If Xj is left-removable in vr — {X \ {xj}), set ej = —. Otherwise Xj must be right-removable 
in vr — (X \ {xj}) and we set ej = +. Suppose to the contrary that no Xj is GT-reducible, 
and thus for every j G [r] there is some vj € W{B;n,7r — Xj) so that Wj = Lw,Xj{vj) contains a 
pattern from B. In fact, Propositions 3 and 5 show that we may assume |u)j| < n -|- ||i3||oo — 1- 

Each copy of a pattern from B in wj must use the entry Xj since dx {wj ) = vj is S-avoiding. 
Hence at most ||-B||oo different Xj's may share the same Wj. Therefore it would suffice to 
show that the number of Wj^s is bounded by some constant depending only on the set B. To 
accomplish this we will show that each Wj lies in the set 

° 4'-x„x. ° • • • ° 4'-~..i-...-.._„x.(W^(^; n,7r- X)), 
by showing that for all j € [r] , 

in,Xj = O 4-Xl,X2 ° • • • ° Xr-UXr ° Qxr ° ' ' ' ° <9^, + l ° Qx,^^ ° ' ' ' ° 

as maps from W{B; n, vr — xj) to VK(0; n, vr). Proposition 3 implies that this would follow from 
/•^i 0/^2 o • • • o / ^'' = 

^■K,Xi "tt — Xl,X2 n~Xl Xr~l,Xr 



7,Xj ^ ^-IV — Xj,Xl O ' ' ' O ^TT — Xl Xj — 2—^j,^j-l ° ^TT — Xl Xj,Xj^l ^ " " " ^ I^^y — X^ Xr-l,Xr1 



and this follows from Remark 4 because the Xj's are pairwise nonadjacent. 

Theorem 2 gives a bound on the number of children a node in T{B) may have. Let A denote 
this bound. Since W{B; n, vr — X) = T{B; st(7r — X)), the number of nodes in W{B; n, vr — X) 
of height between and ||-B||oo — 1 is bounded by 1 -|- A -|- A^ -|- • • • -|- AII^II°°~^. Therefore, 
since we have shown that each wj lies in the image of this set under the map o 4^_^,^ ^2 ° 
■ ■ ■ o i^lj.^ Xr-i xr^ have bounded the number of possible w/s, completing the proof. □ 

Theorem 7 only applies when the set of forbidden patterns B is finite. It can be shown 
that this hypothesis is necessary. First note that if /?i,/32 € B where P2 contains a Pi pattern 
then T{B) = T{B \ {(32}), or in other words, the (32 restriction is superfluous. Thus we may 
always assume that B = min(i?), where here min(i?) denotes the minimal elements of B with 
respect to the pattern containment ordering. It is not immediately obvious that there are sets 
of permutations B such that mia[B) is not finite. Equivalently: are there infinite antichains 
of permutations? Indeed, as has been rediscovered numerous times during the past thirty-five 
years, there are. The reader is referred to Atkinson, Murphy, and Ruskuc [2] for constructions 
and references to earlier work. 

In particular, Av(321,4123) contains an infinite antichain. Following the argument given in 
Murphy's thesis [26] one can show that there is a set of the form Av(-B) where 321,4123 G B 
that does not have a rational generating function. In more detail, there is an infinite antichain 
U C Av(321,4123) with at most one member of each length. If Ui and U2 are two different 
subsets of U then Av(321,4123, Ui) and Av(321,4123, U2) have different generating functions. 
Because U is infinite, this gives uncountably many distinct generating functions, and thus they 
cannot all be rational. In particular, these trees cannot all be finitely labeled. 



5. Conclusion 

The algorithm presented here is implemented in the Maple package FinLabel available at 
http : //math . rutgers . edu/~vatter/. This algorithm is — up to symmetry — only a special 
case of the algorithm of Zeilberger [37]. More precisely, if FinLabel can find a generating 
tree isomorphic to T{B) then Zeilberger's package Wilf can enumerate -B~^-avoiding permu- 
tations. Our algorithm has the advantage that, in the cases that it can handle, it returns both 
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a generating tree and generating function, whereas Zeilberger's algorithm only returns a poly- 
nomial time algorithm for computing | Av„(i?)|. On the other hand, Zeilberger's algorithm is 
applicable in many other situations. For example, it can enumerate the 1234-avoiding permu- 
tations, which are known to have a non-algebraic generating function (Gessel [19]). Zeilberger's 
algorithm is extended in Vatter [33]. 

Since the writing of this paper, the enumerative implication of Theorem 7 has been gener- 
alized by Albert, Linton, and Ruskuc [1]. They introduce a correspondence between permu- 
tations and words called the insertion encoding and prove that if B satisfies the hypotheses 
of Theorem 7 (in fact, weaker hypotheses suffice for their theorem) then the set of insertion 
encodings of permutations from Av{B) forms a regular language. It then follows from the 
theory of formal languages that Av{B) has a rational generating function. 

We conclude with several results that can now be proved completely automatically using the 
FinLabel package. First we have results from the classical paper of Simion and Schmidt [28]: 



B 


generating function for Av(_B) 


{123,213} 


X 

l-2x 


{123, 231} 


(x-iy-i 


{123, 321} 




{132, 231} 


X 

l-2x 


{312,231} 


X 

l-2x 


{123, 132,213} 


x{l-\-x) 
\~x-—x'-^ 


{123,132,231} 


{x~iy^ 


{123, 132,321} 


x + 2x^ + 3x^ + a;* 


{123, 231,312} 


{x~iy^ 


{132,213,231} 


(x~l)-^ 



West [36] undertook a systematic study of permutations that avoid one pattern of length 
three and another of length four. The generating functions that our algorithm can rederive 
are listed in the chart below. 



B 


generating function for Av(_B) 


{123,3214} 


3^(1 — 3;) 


1-Zx + x'-' 


{123,3241} 


x{l-3x+Ax'^ -x'^) 


{2x-\)(l-x)'-^ 


{123,3421} 


x{l-3x + bx'^ -2x^) 


{l-xy- 


{123,4321} 


x + 2x^ + bx^ + 12,x-^ + 25x'5 + 25x'^ 


{132,3214} 


x(l-2x+2x'^) 


1 — 4a; + 5a;2— 3x3 


{132,3241} 


xi\ — x) 


1-Zx + x'^ 


{132,3421} 


a:(l-3a:+3a:^) 


{l-x)(2x-l)'^ 


{132,4321} 


a; (1-331 + 53:^ -21^+3:*) 


(l-x)=> 


{213, 1234} 


x{\ — x) 


l-Zx + x'-^ 


{213, 1243} 


x{l — x) 


l-Zx-\-x^ 


{213, 1423} 


x{l — x) 


l-Zx+x'-^ 


{213,4123} 


x{\ — x) 


l-Zx-\-x'^ 
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Finally we have permutations that avoid two patterns of length four. The following generat- 
ing functions, recently computed by Kremer and Shiu [22], can also be found using FinLabel. 



±> 


generating function for Ay{B) 


{1234, 3214} 


x[l — -ix) 




(x-l)(4,x-l) 




{1234, 3241} 


x(l- lla; + 54a:^- 151x^+2682:'' -313a;^+234i'^ - 


-lOSx'' +29x^ -Ax^) 


{l-3x + x'-'){2x-l)'-'{x-l)'' 




{1234, 3421} 


x{l — 7x+2Ax'^—AAx^+Q2x^—Z9x^+Z2x^ — \9x'^ 


+Ax^) 


\L X) 






X + Zx + bx + IZX + OOX + oubx 


+ oozx +l/t)4j: +1/D4X 




x{l-Ax+5x'^ -ix^)(\-x) 




l-7x+17x'-' -22x'-' +13x-i -Ax^ 




{1243, 3241} 


x(l-9x+31x'^-A9x^+37x''--14x^+2x^) 




(l-3:)(l-4x + 2x^)(l-3x + x^)^ 




{1243, 3421} 


2!(l-9x+34x2-64a;3+642;''-282!^+43;'') 




(a:-l){2a:-l)'' 




{1423,3214} 


a;(l-6i + 12a;2-7a;3 + 2a;*) 




l-8a: + 22a;2-25a:3+10a;4-2a;5 




{1423, 3241} 


a;(2a:-l)^(l-a:) 




l-Tx + Wx'-' -16x-^ +Ax* 




{3214,4123} 


x(l — 3x) 




(x-l)(Ax-l) 
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